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ABSTRACT 



The Carl D. Perkins Vocational and Technical Education Act 
(Perkins III) directs state and local education providers to improve 
vocational and technical education programs. Most of these improvements are 
aimed directly at changing what teachers do and what and how students learn. 
The National Assessment of Vocational Education (NAVE) is charged with 
evaluating the impact of Perkins III and making a report to Congress by July 
2002 . Of chief interest to Congress is how these recommendations for program 
improvement are realized in high schools and how they affect what goes on in 
classrooms. This paper, written to inform NAVE, argues that to understand 
the quality and effects of curricular and teaching changes undertaken in 
response to Perkins III, the evaluation must focus on teachers and 
classrooms. This paper has three goals: (1) to examine the principal themes 

and issues related to assessing quality vocational education (with an 
emphasis on teachers and teaching) ; (2) to lay out a research plan to assess 

quality practice; and (3) to discuss the conceptual, methodological, and 
practical challenges to conducting the research. (YLB) 
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Like its predecessor, the Carl D. Perkins Vocational and Technical Education Act (Perkins IE) directs state and 
local education providers to improve vocational and technical education programs. Most of these 
improvements are aimed directly at changing what teachers do and what and how students learn. The National 
Assessment of Vocational Education (NAVE) is charged with evaluating the impact of Perkins III and making 
a report to Congress by July 2002. Of chief interest to Congress is how these recommendations for program 
improvement are realized in high schools and how they affect what goes on in classrooms. 

This paper, written to inform the design of NAVE, argues that to understand the quality and effects of 
curricular and teaching changes undertaken in response to Perkins III, the evaluation must focus on teachers 
and classrooms. This paper has three goals: (1) to examine the principal themes and issues related to assessing 
quality of vocational education (with an emphasis on teachers and teaching); (2) to lay out a research plan to 
assess quality practice; and (3) to discuss the conceptual, methodological, and practical challenges to 
conducting the research. 

Themes and Issues 

Perkins in promotes several program improvements that are intended to raise the overall quality of vocational 
and technical education programs in high school. The core program improvements address two main areas: (1) 
the content and delivery of vocational programs and (2) the training and staff development of teachers, 
administrators, and others in support of program delivery. The improvements in content and delivery 
emphasize strengthening the ties between school and work by integrating academic and vocational education, 
teaching "all aspects" of an industry, and providing students with skills to utilize technologies in the 
information age. 

Perkins HI also wisely recognizes that those engaged in these integration-related reforms, both in and out of 
school, may need specialized training and professional development to implement them. Teachers certainly 
need support for integration, since they bear most of the responsibility for redesigning curricula and classroom 
activities in support of student learning. Because integration-related reforms require closer connections with the 
workplace, employers are more likely to become involved in school reform efforts. Teachers and school 
administrators need to explore ways to effectively engage employers as reform partners; employers need to 
explore how to partner with schools. Appropriate professional development activities can assist in supporting 
those involved in the design and delivery of vocational programs to define and develop their new roles. 

The program improvements related to content and delivery of vocational education carry over from the 1990 
amendments to the Act (Perkins n). The extent and success of integration was assessed in the previous NAVE, 
which found notable increases in such efforts between 1992 and 1993. However, the evaluation also concluded 
that reforms varied widely in quality, size, and kind and included everything from simply relabeling existing 
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programs and making ad hoc changes to well-planned restructuring efforts (NAVE 1994). While a few states 
had adopted comprehensive reforms of work-related education as a matter of policy, most school systems were 
attempting to fit the Perkins II reforms into existing curricula, rather than making broad changes. At the time of 
the previous NAVE, systematic evaluations of programs or curriculum in support of integration were few and 
often flawed, and, since few programs were actually in place at the time, the research focused on 
implementation processes rather than student outcomes (Stasz, Kaganoff, and Eden 1994). 

In current practice, the content and delivery of vocational and technical education goes way beyond the specific 
issues mentioned in the legislation. Like Perkins n, the guidance provided by Perkins in remains somewhat 
fuzzy, a condition that encouraged a variety of programs to proliferate, with much experimentation and 
innovation (Stasz and Grubb 1991). For example, integration can occur at several levels, from relatively 
straightforward attempts to enhance the academics within vocational courses or to align a sequence of courses 
linked to an occupational area, to much more complex models that affect how a high school is structured (e.g., 
career academies with blocked scheduling; organization around career clusters instead of subject-matter 
departments). With the implementation of the School-to-Work Opportunities Act of 1994 (STWOA), the 
opportunities for high school students to participate in programs with a dual purpose of teaching academic and 
work-related skills and competencies has significantly increased. STWOA enhances existing programs or 
fosters the development of new programs which purposely seek to attract all students, not just those who 
traditionally enroll in vocational education. In addition to integrating academic and vocational education within 
school programs (through course sequencing, team teaching, career academies, career clusters, or majors), 
school programs increasingly promote integration through out-of-school activities, such as job shadowing, 
mentoring, or work-based learning (Hershey, Hudis, Silverberg, and Haimson 1997). 

The expansion of work-related programs and courses through Perkins and related legislation, and the success of 
these programs in attracting new students create several problems for evaluators. First, it is difficult to identify 
the intervention. Programs calling themselves "integrated" vary widely in terms the extent of curricular 
integration they provide and the type of activities that students engage in, both in and outside the classroom. 
Each program can have multiple purposes and goals that shape the curriculum and instructional offerings. As a 
result, there is likely to be as much variation within programs of a certain type as between them. Many new 
programs, especially those developed through STWOA, may not be exclusively supported with Perkins funds 
and may not even be identified as vocational programs. Likewise, it may be difficult to identify participants as 
vocational students. However, the extent and success of these programs are surely of interest to Congress and 
other policymakers whether or not they flow directly from Perkins dollars or serve the traditional vocational 
student. 

As has been mentioned, integration-related curricular reforms are embodied in various types of program 
models (Grubb, Davis, Lum, Plihal, and Morgaine 1991). It is within these models that changes in instructional 
practice — what is taught and how it is taught — take place. The bulk of the research on integration-related 
reforms to date provides descriptive data on the various forms of integration and the various challenges that 
education providers face in implementing integration-related reforms (e.g., Bodilly, Ramsey, Stasz, and Eden 
1993; Schmidt, Finch, Faulkner, Isom, Magee, and Fox 1992a, 1992b). Relatively few studies, however, have 
investigated integration at the level of instructional practice, either by carefully documenting changes in the 
curriculum or by describing how teachers organize instructional activities. Few studies attempt to link practices 
with outcomes. Thus, relatively little is known about the quality of integration-related reforms at the classroom 
level that enables one to identify "best practices" for promoting desired student attainments. 

The failure to identify best practice in vocational education is due more to the state of the research than to lack 
of understanding. One of the chief reasons given for integrating academic and vocational education is 
pedagogical: integration should promote instructional practices that emphasize the application of abstract or 
theoretical concepts (the academic) to real, practical problems (the vocational). The move to contextualized or 
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applied or experiential approaches to teaching and learning has been developing from research on how people 
learn. Since at least the late 1980s, scholars explicitly use traditional vocational apprenticeship as a model for 
contextualized learning (e.g., Raizen 1989; Collins, Brown, and Newman 1989; Lave 1988). Others draw 
insights from studies of learning at work or from problem solving in everyday life to illustrate various 
shortcomings of school-based learning for helping students make use of formal knowledge (e.g., Resnick 1987; 
Scribner, 1982). As belief in the value of contextualized or applied pedagogies increases, the effect is to bring 
this central aspect of vocational education teaching into the mainstream of thought concerning "best" practice. 
While the theoretical work is suggestive, there is little systematic research on the effectiveness of these 
approaches, especially with respect to meeting academic standards. 

The form and manner of integration-related activities are largely in the hands of school-based practitioners. 
Perkins III rightly emphasizes the need to support teachers and other school personnel with appropriate 
professional development. Teacher quality is central to the success of any initiative aimed at instructional 
practice, so it will be important to consider what preparation they have (preservice education, industry 
experience) and what kinds of professional development they need. Previous research suggests that few 
teachers are prepared to meet the challenges of creating curricula or altering their teaching to promote 
integration (e.g., Bodilly et al. 1993). Even when prepared or motivated, they are not well-supported in their 
efforts through such actions as shared planning time or other incentives. Studies of integration-related reforms 
spell out what kinds of professional development teachers need or propose inservice training models that will 

help teachers use constructivist teaching methods and develop integrated curricula (Stasz, 1997) 2 A1 though the 
opportunities for teachers to participate in staff development geared toward integration are on the rise, they are 
by no means common practice for most teachers in these programs. While the need for professional 
development for employers or administrators is recognized, much less work has been done to design or deliver 
appropriate activities for nonteachers. 

The more ambitious attempts to integrate academic and vocational education require collaboration with 
employers. An important issue for NAVE concerns linkages with the employer community that help improve 
the quality of vocational education at many levels. The growth of work-based learning connected to school has 
focused research on employer participation, and the literature describes which employers participate and for 
what reasons, and how the education community can promote participation (e.g., Bailey, Hughes, and Barr, 
1998). By and large, few employers participate in ways that directly affect curriculum and teaching in 
classrooms, nor is it clear whether such participation is feasible or desirable, at least at the high school level. 
Employers may have an even more important role to play, however, as industry skill standards continue to be 
developed and used as quality benchmarks for designing curricula. The federal government is supporting over 
twenty projects to develop industry skill standards, and NAVE provides an excellent opportunity to determine 

whether and how these efforts impact on high school programs. 2 

The legislative directives, coupled with the state of integration in practice, suggest a research approach with the 
following characteristics: 

• A focus on teachers and teaching that can describe integration at the level of classroom practice and 
identify relationships between teacher characteristics, instructional practices, and student attainments; 

• An emphasis on program types that have been less studied and potentially serve to restructure the whole 
school, such as career clusters or career major strategies. Such strategies are especially important because 
they serve to connect vocational education to other school reforms, rather than remaining on the 

periphery of the high school program.^ 

• Careful attention to previous or ongoing studies on which the current assessment might build or extend. 



The following section outlines a research strategy for assessing the quality of vocational education in high 
schools that incorporates these principles and the main issues discussed thus far. 

A Research Strategy 

Previous NAVE evaluations primarily utilized survey methods to examine the extent of integration or other 
specific program improvements discussed in the legislation. This is certainly an acceptable method for 
understanding whether a particular reform is widespread, but provides little information about how reform 
policies are specifically realized in everyday curriculum content or teaching methods. Although surveys will 
certainly have an important place in NAVE, a case study approach is more appropriate for examining how 
vocational education reforms affect classroom practices. Ideally, case studies of practice should be linked to 
quality standards and student outcomes, although this may not be possible given time and data limitations 
(discussed further below). What follows is a sketch of a study design, which can address several questions of 

interest, such as: 

• What are the general characteristics of programs that integrate academic and vocational education? What 
are the goals of the program and how do they relate to the school as a whole? 

• How does the curriculum support desired improvements (e.g., integration, "all aspects" of the industry, 
technology-related skills). Does the curriculum align with appropriate quality benchmarks, such as 
graduation requirements or academic or industry skill standards? 

• What teaching practices are evident in these programs? What kinds of activities do students engage in? 
Do the activities enhance contextualized/applied learning of academic and vocational subject matter 
(e.g., instruction anchored in and learning grows out of authentic problems, which lead to authentic 

products for authentic audiences^)? 

• What do students learn in these programs? What is the quality of the work they produce? How do they 
perceive the learning experience? What kinds of student performance data are collected and how are they 
used? 

• What are the characteristics of participating teachers (e.g., educational background, work experience)? 
What kinds of staff development activities have they engaged in to support their teaching? What kinds of 
further support do they need? 

• What are the roles of administrators and employers? How can they be better supported to achieve 
program improvement goals? 

• How does the program relate to district or statewide reforms, including standards and assessments or 
core performance indicators? 

A replicated case study design is most appropriate because the interest is in examining and interpreting ongoing 
processes in real-world contexts, especially where the processes to be studied (teaching, learning) are not easily 
separable from the context and where the variables of interest outnumber the units of study (Yin 1994). In this 
design, similar sets of criteria are used to select sites and the individuals within them and common 
data-gathering instruments and procedures are used across sites. Because classroom practices are influenced by 
the broader institutional context, it is important to gather data at multiple levels (e.g., the classroom, school, 
district and state) and from a variety of informants (policymakers, school administrators, employers, teachers, 
and students). 
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Consider a study design focused on a purposive sample of 20 to 40 teachers and classrooms. Different 
sampling frames are feasible, but one approach is to select three states, three schools within each, and 2—3 
teachers within each school for intensive study. The states and schools should represent serious efforts to recast 
vocational education along the lines recommended in the legislation. In addition, states which can provide data 
on student attainments (e.g., achievement tests, UI data for tracking post-high school attainments) or teacher 
quality (e.g., personnel records), and which are agreeable to sharing the data, are preferable. The school sample 
should include program forms that have been less studied, such as career majors or cluster arrangements. 

Within schools, programs and teachers should be selected with some thought to subject area, perhaps focusing 
on two career areas (e.g., health and technology/electronics). 6 Participating teachers can be randomly selected 
from the targeted school/program. The overall intent is to draw a purposive sample where the conditions for 
creating and sustaining desirable practice are in place. 

By nesting the sample of teachers and classrooms within programs, schools, and states, it will be possible to 
examine study questions at several levels and to track the relationships to classroom practices (both curriculum 
and instruction) or to teacher quality. The study design should utilize multiple data gathering activities and 
sources of data. For example: 

• Semi-structnred interviews of state, district, school level administrators and policymakers, employers to 
examine various policy and implementation issues. Semi-structured interviews can gather core data from 
different respondents, yet also permit tailoring interview questions to specific situations and respondents 
and allow the interviewer to pursue useful lines of inquiry that may not be known prior to site visits. 

• Teacher surveys to include target teachers, plus a larger sample of teachers within the same school. The 
survey would focus on teacher background and staff development activities, curriculum, and teaching 

practices.-^ 

• Classroom observations . Assuming a two-year time frame, two observations per year, at least one of 

which should be videotaped^. Observations should be completely open-ended, with fieldnotes 
transcribed. Several computer-based programs for working with text-based data are available for coding 
and analysis. Observations are the best means to gather information on teaching practice. 

• Teacher activity logs for selected time periods can gather information (through closed and open-ended 
items) about specific lessons, including teacher objectives, student activities, curriculum content and 
teaching behaviors. Teacher assessments, quizzes, or homework assignments in connection with these 
lessons can also be used to gauge quality. 

• Student transcripts and surveys would gather information on student coursetaking and perceptions of 
curriculum and teaching. 

• Student work. Examples of student work in the case-study classrooms would provide important 
information about the quality of the activities, level of difficulty, and extent of contextualized learning. 

• Student records . Ideally, the study would include some measure of student attainment, either in school or 
post-school. Although the time frame is short, it will be important for NAVE to explore possible options, 
e.g., pre-post testing or analysis of available test data from school, district, or state. 

• Documents . Existing documents, such as state curriculum frameworks, course syllabi, industry skill 
standards, or policy documents will complement interviews or other data. 

A general approach to analyzing the qualitative data is to begin with an a priori set of theoretical or conceptual 



themes that define key questions, variables, and expected relationships. The data-gathering methods discussed 
above should align with study questions and themes. Data analysis would follow an iterative process that 
begins by coding/organizing data by themes, but permits new themes or issues to arise. For example, the 
analysis of observational/video and log data might follow current theoretical and conceptual frameworks suited 
to describing contextualized teaching and learning approaches. The curriculum can be described along several 
dimensions, such as cognitive complexity, academic and vocational content, and adherence to accepted 
curriculum or industry standards. The analysis might proceed case by case, followed by cross-case analysis to 
identify any general findings. 

Connections to Other Work 

NAVE should explore ways to relate the study of teaching practices sketched here with recent or ongoing 
studies. For example, Bob Crain and his colleagues at teachers’ college recently completed a longitudinal study 
of career magnet high schools (Crain, Allen, Little, Sullivan, Thaler, Quigley, and Zellman 1999). It may be 
possible to extend this work by following up on student attainments post-high school. A study of career 
academy programs being conducted by Jim Kemple and others at MDRC (Kemple, Poglinco, and Snipes 1999) 
may provide information about successful programs that could be further studied in NAVE. At LRDC, Lauren 
Resnick is leading a study on contextualized learning approaches for learning academic subjects. This project 
will likely yield both a literature review and some useful conceptual frameworks or study sites for future work. 

The last NAVE commissioned a literature review on integrating academic and vocational education (Stasz, 
Kaganoff, and Eden 1994). Given the proliferation of programs and studies in the last five years, a review 
which focuses on quality and effectiveness of integration-related programs (rather than descriptions of 
implementation processes) would be most useful at this time. This review should be quite broad and include 
case studies of teaching practices as well as those using longitudinal data. David Stem and others, for example, 
are currently looking at participation in vocational and school-to-work programs and various student outcomes, 
using the NLSY97. 

Difficulties and Limitations 

Any evaluation of the quality of vocational education in high schools that is sensitive to issues of teachers and 
teaching practice faces a number of significant challenges. 

• How to define "quality?" Program quality can be identified along several dimensions, such as student ' 
outcomes, adherence to a set of accepted curriculum or industry standards, or adherence to some 
accepted model of contextualized learning. It is unlikely that student outcome data can be used to 
identify "successful" sites, because comparable data suitable for selecting programs in multiple states are 
simply not available. Therefore, decisions about "quality" (either for selecting programs/classrooms to 
study or for characterizing the teaching and learning that is observed in them) must be left to expert 
judgment following some conceptual or theoretical framework. 

• How to measure student outcomes or learning? A pre-post design within sampled classes and with 
nonparticipating students might be possible, if technical issues could be overcome.^ It may be possible to 
administer a standardized test such as Work Keys, which assesses applied academics (e.g., applied math, 
reading for information, applied technology) and other employability skills (e.g., teamwork, motivation, 

managing resources)^. Developing these types of skills are likely to be a concern of most 
integration-related programs. Within-school comparisons of selected student outcomes for participating 
and nonparticipating students are also feasible (e.g., GPA, attendance, credit pace, and other data 
regularly collected by high schools), but are problematic due to selection effects or other potential 
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sources of bias.^- 



• How to define "vocational" programs and students? The past decade has seen many important 
innovations in the design of high school programs that link school and work. The aim of Perkins II and 
other federal reforms has been to blur the lines between academic and vocational courses, students, or 
teachers. Thus, it is difficult to identify strictly vocational programs or students or to determine what 
vocational education’s unique contribution to student learning. 

• How to measure instructional practice? The difficulties inherent in assessing instructional practice are 
well documented. 12 Generally, survey data are most useful for gathering information about curriculum or 
course taking if the standard is knowing whether or not a topic has been taught and if it has been taught 
over several periods or weeks. Surveys are insufficient for gathering reliable data on teaching practice, 
especially when practice may be in transition. This is certainly the case for identifying contextualized 
teaching practices. When practice is in flux, nonsurvey data (teacher logs, interviews, observations) and 
the use of exams, quizzes, assignments, and student work provide a richer picture of classroom practices. 
These nonsurvey methods are, of course, more expensive to gather and analyze and present other 
difficulties in interpretation and inference. 

• How to measure teacher quality? Studies of teacher quality typically relate specific background 
characteristics (e.g., undergraduate training, certification, and scores on ability tests) with various student 
outcomes (e.g., Ehrenberg, Goldhaber, and Brewer 1995; Goldhaber and Brewer 1997). This line of 
research finds that some characteristics, such as teachers’ educational background, are associated with 
higher student attainments. These studies do not link quality measures with teaching practices, however, 
so how practice relates to teacher characteristics is largely unknown. Within the NAVE time frame, it 
will be possible to gather information on teacher characteristics through self-report surveys and to relate 
some background characteristics with practices in a limited way. However, the study outlined here could 
not address supply and demand of vocational teachers or other labor market issues that are currently of 
great interest to policymakers. As mentioned earlier, the latter type of analysis requires detailed 
personnel data that few states systematically gather. 

• Are relationships between teaching practice and student outcomes generalizable? Determining statistical 
general izibility requires data on a large number of teachers and students. This creates an obvious 
dilemma, because survey data, which would provide information on enough teachers, is by itself 
inadequate for assessing teaching practice. A case study approach, as described above, will provide more 
valid information, but is limited for determining statistical generalizibility. However, in order to get a 
sense of how integration-related reforms are realized in high schools at a level of detail that is useful to 
teachers, it is most sensible to pursue an approach that yields specific information about teaching 
practice — e.g., about the teaching strategies and tactics, approaches for designing integrated curricula and 
complimentary assessments, and the principles for creating effective learning activities. 

The approach outlined here also has several important advantages. First, the inclusion of video observations in 
the case study design will yield an important contribution to research on teaching practice. Newer technologies 
make video data and analysis affordable, and, most importantly, make it possible to create a database that has 
use beyond NAVE. It is also quite feasible to obtain the necessary releases to use the videos for further 
research.^ 2 The study would also help advance understanding of contextualized or applied teaching approaches 
that many believe will have most significant impact on student learning. 

Video examples of effective teaching in integration-related programs would be very useful for staff 
development purposes, as they would provide specific models for teachers to observe.^ Presently, research 
suggests that there are many significant barriers to implementing integration-related reforms, several of which 
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relate to teacher staff development. Some of these barriers are administrative, such as lack of time during the 
school day for teachers to collaborate to develop appropriate curricula. Many teachers lack explicit training in 
contextualized learning approaches and typical short-term staff development activities are often inadequate for 
providing the skills they need. Most teachers lack the training and skills to develop assessments that are 
appropriate for use in contextualized learning environments (Stasz 1997). 

In sum, due primarily to time and data limitations, it is not feasible for NAVE to reliably assess teaching 
practice linked to quality standards and student outcomes. However, a replicated case study design that closely 
aligns multiple data sources and methods with selected questions of interest can provide new information about 
teachers, curriculum and instructional practices in programs that explicitly connect school and work. If the 
evaluation includes another wave of survey data, it will be possible to gather information on curriculum, 
teaching practice and student characteristics from a larger sample of teachers, as well as track the extent of 
implementation since the last NAVE. An updated literature review can both inform the analysis of the case 
studies, as well as summarize findings on the extent and success of integration -related reforms. By coordinating 
case study and survey methods, NAVE should be able to provide a useful picture of the quality of vocational 
education in high schools today. 
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